Performance test of software products with reduced duration

ABSTRACT

A method and system for testing performance of a software product. One or more complete executions of a performance test of corresponding versions of the software product are performed. The complete executions of the performance test identifies a steady state value of each performance parameter monitored during the performance test after each performance parameter has transitioned to a steady state during the performance test. A performance profile of the software product is determined according to the complete executions of the performance test. Partial executions of the performance test of corresponding further versions of the software product are performed, each partial execution having a partial duration preceding the steady state of each performance parameter. Corresponding results of the partial executions of the performance test are estimated according to the performance profile from corresponding partial trends of each performance parameter during the partial executions of the performance test.

TECHNICAL FIELD

The present invention relates generally to the information technologyfield, and more specifically, to performance tests of software products.

BACKGROUND

Verification of correct operation of any software products is a verycritical activity in any phase of their lifecycle. Particularly, in manysituations it is important to verify that the software products performas expected, for example, in terms of consumption of hardware and/orsoftware resources (such as processing power) by the software products,which is very useful for several purposes such as, for example, tovalidate the software products, to predict the behavior of the softwareproducts in difference scenarios.

Some techniques have been proposed for estimating performance ofsoftware products. For example, a complexity analysis may be applied toany software product in an attempt to infer performance of the softwareproduct therefrom. Moreover, a contribution to an overall performance ofthe software product by different components thereof may be predicted byanalyzing the effects of artificial performance degradations insertedinto the software product.

A more accurate verification of performance of any software product isgenerally obtained by performing a corresponding performance test, sothat one or more performance parameters of the software product (forexample, its processing power consumption) are actually measured byrunning the software product in one or more reference scenarios.

SUMMARY

Embodiments of the present invention provide a method, and associatedcomputer system and computer program product, for testing performance ofa software product.

One or more processors of the computer system perform one or morecomplete executions of a performance test of corresponding versions ofthe software product. The complete executions of the performance testidentify a steady state value of each performance parameter of at leastone performance parameter monitored during the performance test aftereach performance parameter has transitioned to a steady state during theperformance test,

The one or more processors determine a performance profile of thesoftware product according to the complete executions of the performancetest. The performance profile models a complete trend of the at leastone performance parameter during the complete executions of theperformance test.

The one or more processors perform one or more partial executions of theperformance test of corresponding further versions of the softwareproduct. Each partial execution of the performance test has a partialduration preceding the steady state of each performance parameter.

The one or more processors estimate corresponding results of the partialexecutions of the performance test according to the performance profilefrom corresponding partial trends of the at least one performanceparameter during the partial executions of the performance test.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1D show the general principles of embodiments of the presentinvention.

FIG. 2 shows a schematic block diagram of a computing system or computerin which embodiments of the present invention may be practiced.

FIG. 3 shows software components that may be used to implementembodiments of the present invention.

FIGS. 4A-4B show an activity diagram describing the flow of activitiespertaining to implementation of embodiments of the present invention.

DETAILED DESCRIPTION

Execution of a performance test is relatively complex. Therefore, sometechniques have been proposed aimed at simplifying the performance test.For example, when the performance test comprises repeated iterations (toprovide results that are significant on the average), a test criteriamay be used to limit the number of iterations.

However, each iteration of the performance test may be quite timeconsuming. Indeed, the performance test generally starts with atransient phase wherein the performance parameters change over time(because of a change of state caused by the start of the softwareproduct). For example, during the transient phase the performanceparameters increase or decrease continuously (for example, with theprocessing power consumption that starts from a relatively high valuecaused by the operations required to initialize the software product andthen decreases as these operations complete). At the same time, theperformance parameters may have spikes; i.e., fast and short positive ornegative peaks (for example, with the processing power consumption thatincreases briefly to perform specific operations). The values of theperformance parameters in the transient phase are scarcely significantand are generally discarded. After some time, the performance testshould reach a steady state wherein the performance parameters aresubstantially stationary; i.e., the values of the performance parametersfluctuate around a constant value (for example, with the processingpower consumption that stabilizes to a value corresponding to theworkload of the software product). The values of the performanceparameters in the steady state represent the actual behavior of thesoftware product, and then define the result of the performance test (incomparison with the performance test that instead fails if the steadystate is not reached).

However, the recognition of the steady state (and then the determinationof the result of the performance test) requires verifying that acorresponding stability condition (defined by the performance parametersremaining stationary) is fulfilled in a stability time-window longenough to establish the actual end of the transient phase with asufficient degree of confidence. As a result, the overall duration ofthe performance test (given by the transient phase plus the stabilitytime-window) is relatively long (for example, of the order of severalhours or days).

As a consequence, a development time of the software product increasesaccordingly, which may be especially critical in case of a relativelyshort development cycle, wherein new versions (or builds) of thesoftware product should be delivered with high frequency (e.g., up toevery few days).

FIGS. 1A-ID show the general principles of embodiments of the presentinvention.

Starting from FIG. 1A, performance of a software product is tested (fordifferent versions of the software product that are delivered overtime). At the beginning, one or more complete executions of aperformance test (of corresponding versions of the software product) areperformed. The complete executions of the performance test providecorresponding results. Each result comprises one or more values of aperformance parameter after a steady state thereof has been recognized(at the end of a transitory phase), which happens, for example, when acorresponding stability condition (defined by the performance parameterthat remains stationary) is fulfilled in a corresponding stabilitytime-window (for example, requiring 20-30 hours from the beginning ofthe performance test).

Moving to FIG. 1B, a performance profile of the software product isdetermined according to the complete executions of the performance test.The performance profile models a (complete) trend of each performanceparameter during the complete executions of the performance test (withparticular reference to the transient phase of the performance test).For example, as shown schematically in FIG. 1B, at the beginning thecomplete trend increases over time (during the transient phase) and thenremains stationary at a constant value (in the steady state). The resultof the performance test is given by the values of the performanceparameters in the steady state, which may be determined only after thestability time-window from the end of the transient phase (once thesteady state has been recognized by the fulfillment of the stabilitycondition).

Moving to FIG. 1C, one or more partial executions of the performancetest (of corresponding versions of the software product) are nowperformed. Each partial execution of the performance test has a(partial) duration that precedes the recognition of the steady state ofthe performance parameter (for example, 5-10 hours from the beginning ofthe performance test). At the same time, a (partial) trend of theperformance parameter is determined (for example, by low-pass filteringthe values of the performance parameter that are measured during thepartial execution of the performance test).

Moving to FIG. 1D, the result of each partial execution of theperformance test is estimated from the partial trend of the performancetest, according to the performance profile. For example, it is possibleto assume that the partial trend would have continued in the same way asthe performance profile. The value of the performance parametercorresponding to the steady state of the performance profile thendefines the result of the performance test.

The above-described solution significantly shortens the duration of theperformance test. Indeed, the result of the performance test may beestimated without waiting for completion of the performance test (i.e.,the recognition of the steady state by the fulfillment of the stabilitycondition). For example, it is possible to stop the performance testshortly after the partial trend has become stationary (withoutcompleting the stability time-window) or even when the partial trend isstill increasing/decreasing.

As a consequence, a development time of the software product reducesaccordingly, which is especially important in case of a relatively shortdevelopment cycle by allowing delivering new versions (builds) of thesoftware product with high frequency (e.g., up to every few days).

FIG. 2 shows a schematic block diagram of a computing system or computer200 in which embodiments of the present invention may be practiced.

For example, the computing system or computer 200 may be a PersonalComputer (PC), or a computer. The computer 200 comprises several unitsthat are interconnected through a bus structure 205. Particularly, oneor more microprocessors (μP) 210 control operation of the computer 200.A non-volatile read only memory (ROM) 215 stores basic code for abootstrap of the computer 200 and a volatile random access memory (RAM)220 is used as a working memory by the microprocessors 210. The computer200 is provided with a mass-memory for storing programs and data suchas, for example, a hard disk 225 and a drive 230 for reading/writingremovable storage units 235 (e.g., Digital Versatile Discs (DVDs)).Moreover, the computer 200 comprises a number of peripheral, orInput/Output (I/O), units 240. For example, the peripheral units 240 maycomprise a keyboard and a mouse (for inputting commands and data), amonitor (for displaying information) and a network interface (forconnecting the computer 200 to a communication network such as, forexample, the Internet).

FIG. 3 shows software components that may be used to implementembodiments of the present invention.

All the software components (programs and data) are denoted collectivelywith the reference 300. The software components are typically stored inthe mass memory (e.g., 225, 235) and loaded (at least partially) intothe working memory (e.g., RAM) of the computer when the programs arerunning, together with an operating system and other applicationprograms (not shown in the FIG. 3). The programs are initially installedin the mass memory, for example, from removable storage units or fromthe communication network. In this respect, each program may be amodule, segment or portion of code, which comprises one or moreexecutable instructions for implementing embodiments of the presentinvention.

A test tool 305 is used to execute performance tests of differentversions of one or more software products 310. The test tool 305comprises the following components. A test manager 315 controls theexecution of the performance tests. The test manager 315 accesses (inread mode) the software products 310. Moreover, the test manager 315accesses (in read/write mode) a test case repository 320, which stores atest suite for each software product. The test suite comprises one ormore test cases, each test case defining one or more actions to beperformed on the software product for testing one or more correspondingperformance parameters of the software product in a corresponding(reference) scenario. For example, the performance parameters maymeasure a consumption of hardware and/or software resources of thecomputer by the software product (such as processing power, workingmemory, mass memory), and a responsiveness of the software product (suchas average values of response time, execution time, throughput ofcorresponding transactions). In this way, the performance test ofdifferent versions of each software product are executed in the same(test) conditions; i.e., in the same computer (or type thereof), in thesame external conditions (for example, network connections, concurrentsoftware products) and with the same inputs (as defined by the testsuite). The test manager 315 accesses (in write mode) a performancesignal repository 325, which stores a performance signal for eachperformance test. The performance signal tracks the values of thecorresponding performance parameter that are measured over time duringthe execution of the performance test.

A filter 330 is used to determine a performance trend representing atrend of the performance signal of each performance test. The filter 330accesses (in read mode) the performance signal repository 325. Moreover,the filter 330 accesses (in write mode) a performance trend repository335 which stores an indication of the performance trend of eachperformance parameter (for example, represented by a correspondingparametric function). A monitor 340 is used to monitor the (complete)performance trend of each performance test to recognize the steady stateof the corresponding performance parameter (as soon as the stabilitycondition is fulfilled) and then to determine a result of theperformance test (as defined by the value of the performance trend inthe steady state). The monitor 340 accesses (in read mode) theperformance trend repository 335. Moreover, the monitor 340 accesses (inwrite mode) a result repository 345 which stores the result of eachperformance test that has been executed over time for the differentversions of each software product.

According to an embodiment of the present invention, a modeler 350 isused to determine a performance profile modeling the performance trendsof each software product. The modeler 350 accesses (in read mode) theperformance trend repository 335. Moreover, the modeler 350 accesses (inwrite mode) a performance profile repository 355 which stores theperformance profile of each software product (together with thecorresponding partial duration that has been determined for the partialexecutions of each software product's performance test). An estimator360 is used to estimate the result of each partial execution of theperformance test from the (partial) performance trend of the performancetest according to the corresponding performance profile. Thus, theestimator 360 accesses (in read mode) the performance trend repository335 and the performance profile repository 355. Moreover, the estimator360 accesses (in write mode) the result repository 345.

FIGS. 4A-4B show an activity diagram describing the flow of activitiespertaining to implementation of embodiments of the present invention.

Particularly, FIGS. 4A-4B represent an exemplary process that may beused to perform performance tests of different versions of a genericsoftware product with a method 400. In this respect, each block in FIGS.4A-4B may correspond to one or more executable instructions forimplementing the specified logical function on the computer.

The process passes from block 403 to block 406 as soon as a currentversion of the software program has to be tested (for example, accordingto a corresponding test plan). In response thereto, the test managerdetermines one or more software metrics of the current version of thesoftware product that characterize the software product. The softwaremetrics may be set manually (for example, by a tester that enters themetric's value in the test manager) or automatically. For example, inthe latter case the test manager measures a complexity of the softwareproduct (such as cyclomatic complexity of the software product, numberof classes of the software product) and/or size of the software product(such as number of source code lines of the software product, number ofobject code bytes of the software product). The test manager at block409 then calculates a change indictor that indicates a change of thecurrent version of the software product with respect to a previousversion of the software product (if any). The change indicator iscalculated according to a comparison of the software metrics of thecurrent version with the software metrics of the previous version of thesoftware product, determined in the same way during the performance testof the software product (for example, by an average difference inpercentage in the software metrics). Continuing to block 412, the testmanager retrieves the test suite of the software product (from thecorresponding repository) and then takes a (current) test case thereofinto account (starting from a first test case), which test case definesa performance test for the corresponding one or more performanceparameters.

The flow of activity branches at block 415 according to the type(complete or partial) of the performance test. Particularly, theperformance test is complete for a number of first versions of thesoftware product required to determine the corresponding performanceprofile with an acceptable degree of confidence (for example, 5-10).This operation of the performance test may also be repeated (uponrequest by the tester) for any significant change of the softwareproduct (for example, a new release thereof), such that the processcontinues to block 418 wherein the test manager runs the performancetest (by executing the actions indicated in the test case insuccession).

Periodically (for example, every 10-100 ms) the test manager at block421 measures the (current) value of the one or more performanceparameters to be verified and adds this value to the performance test'sperformance signal (in the corresponding repository). Continuing toblock 424, the filter 330 accordingly updates the corresponding(complete) performance trend (in the corresponding repository). Forexample, at first the filter may smooth the performance signal, in orderto remove (or at least significantly reduce) short and fastpositive/negative spikes therein (for example, by applying a medianfunction). The filter may then apply a maximum or a minimum intensityprojection algorithm to the (smoothed) performance signal (wherein theperformance signal is held at the performance signal's maximum orminimum over time according to whether the performance signal isgenerally increasing or decreasing, respectively, in the transientphase), in order to remove (or at least significantly reduce) any strongvariations thereof. In this way, the desired result is achieved in asimple and fast way.

With reference now to block 427, the monitor 340 verifies whether thestability condition is fulfilled, which happens when the performancetrend remains at the same value in the stability time-windows (forexample, 5-15 hours); i.e., its first derivative of the performancetrend is substantially zero. If not, meaning that the steady state hasnot been recognized yet, the monitor at block 430 verifies whether amaximum duration of the performance test has been reached (for example,50-100 hours). If not, the flow of activity returns to the block 421 torepeat the same operations continually. Conversely, as soon as themaximum duration has been reached (without recognizing the steadystate), the monitor at block 433 sets the performance test as failed inthe result repository (since the software product has become instableand then needs to be fixed). Referring back to the block 427, if thestability condition is fulfilled the steady state is recognized. Inresponse thereto, the monitor at block 436 determines the result of the(passed) performance test as the current value of the performanceparameter in the performance trend, and then saves the current value ofthe performance parameter in the corresponding repository. Continuing toblock 439, the monitor updates the performance profile of the softwareprogram accordingly. For example, the performance profile is determinedas an average of the available performance trends of the softwareproducts (relating to any version thereof that has been tested).

Referring back to the block 415, in the other situations (i.e., nextversions of the software product after determination of thecorresponding performance profile), the performance test is partial. Inthis case, the estimator at block 442 retrieves the partial duration ofthe performance test from the performance profile repository(initialized to the time required for recognizing the steady state inthe performance profile). The estimator at block 445 estimates aperformance target of the performance parameter corresponding to thepartial duration. The performance target is calculated by evaluating theperformance profile at the end of the partial duration, and thenupdating this value of the performance target according to the changeindicator (for example, scaling this value of the performance target bya percentage thereof). Therefore, the performance target represents thevalue of the performance parameter that is expected at the end of thepartial duration, taking into account the executions of the performancetest on the previous versions of the software product and the change ofthe current version of the software product with respect thereto.Continuing to block 448, as above the test manager 315 runs theperformance test (by executing the actions indicated in the test case insuccession). Periodically, the test manager at block 451 measures the(current) value of the performance parameter to be verified and addsthis current value of the performance parameter to the performancetest's performance signal (in the corresponding repository). Continuingto block 454, the filter 330 accordingly updates the corresponding(partial) performance trend (in the corresponding repository).

With reference to block 457, however, the monitor now verifies whetherthe performance trend has reached the performance target (before the endof the partial duration), which, for example, happens when the currentvalue of the performance trend (generally increasing or decreasing)becomes higher or lower, respectively, than the performance target and aremaining time before reaching the end of the partial duration is belowa threshold (for example, 1-2 hours). If so, the monitor at block 460sets the performance test as failed in the result repository. Thus, theperformance trend tends to depart significantly from the performanceprofile, so that it is likely that the current value of the performancetrend at the end of the partial duration will be not significant, whichallows recognizing the failure of the performance test in advance(before the end of the partial duration), thereby saving further time.Moreover, this result is achieved in a very simple way (by a merecomparison of values). Referring back to the block 457, if theperformance trend has not reached the performance target, the monitor atblock 463 verifies whether the end of the partial duration has beenreached. If not, the flow of activity returns to the block 451 to repeatthe same operations continually. Conversely, as soon as the end of thepartial duration has been reached, the estimator at block 466 verifieswhether the performance trend matches the performance target which, forexample, happens when a difference between the current value of theperformance trend and the performance target is lower (in absolutevalue) than a threshold (for example, 1-5% of the performance target).If not, the partial duration might be too short for estimating theresult of the performance test with an acceptable degree of accuracy.Therefore, the estimator at block 469 increases the partial duration bya delta value (for example, 30-60 minutes). The flow of activity thenreturns to the block 451 to continue the performance test (with theexecution of the performance test that returns to be complete, not shownin FIG. 4B, once the partial duration reaches a total duration of theperformance profile).

Referring back to the block 466, if the performance trend matches theperformance target (meaning that it is possible to estimate the resultof the performance test with an acceptable degree of accuracy), theestimator at block 472 verifies whether a minimum value of the partialduration has been reached, which, for example, happens when theimmediately lower value of the partial duration has been found to be tooshort for a predefined number of consecutive times (for example, 1-3),or in any case when a safety value has been reached (for example, 40-60%of a duration of the transient phase in the performance profile). Ifnot, the estimator at block 475 decreases the partial duration in thereference model repository by the same delta value (for a next partialexecution of the performance test). The process then descends into block478. The same point at block 478 is also reached directly from the block472 if the minimum value of the partial duration has already beenreached. At this point, the estimator calculates the values that areestimated for the performance parameter in the steady state according tothe performance profile starting from the current value of theperformance trend (for example, by increasing/decreasing the value ofthe performance profile in the steady state by a difference between thecurrent value of the performance trend and the current value of theperformance profile). Continuing to block 481, the estimator updates theperformance profile as above.

The process flows to block 484 from the block 433, the block 439, theblock 460 or the block 481. At this point (block 484), the test managerverifies whether a last test case has been run. If not, the flow ofactivity returns to the block 412 to repeat the same operations for anext test case. Conversely, once all the test cases have been run(thereby completing the test of the software product), the flow ofactivity returns to the block 403 waiting for a next test.

In order to satisfy local and specific requirements, a person skilled inthe art may apply many logical and/or physical modifications andalterations to the present disclosure. More specifically, although thisdisclosure has been described with a certain degree of particularitywith reference to one or more embodiments thereof, it should beunderstood that various omissions, substitutions and changes in the formand details as well as other embodiments are possible. Particularly,different embodiments of the present disclosure may even be practicedwithout the specific details (such as the numerical values) set forth inthe preceding description to provide a more thorough understandingthereof; conversely, well-known features may have been omitted orsimplified in order not to obscure the description with unnecessaryparticulars. Moreover, it is expressly intended that specific elementsand/or method steps described in connection with any embodiment of thepresent disclosure may be incorporated in any other embodiment as amatter of general design choice. In any case, each numerical valueshould be read as modified by the term about (unless already done) andeach range of numerical values should be intended as expresslyspecifying any possible number along the continuum within the range(comprising its end points). Moreover, ordinal or other qualifiers aremerely used as labels to distinguish elements with the same name but donot by themselves connote any priority, precedence or order. The termsinclude, comprise, have, contain and involve (and any forms thereof)should be intended with an open, non-exhaustive meaning (i.e., notlimited to the recited items), the terms based on, dependent on,according to, function of (and any forms thereof) should be intended asa non-exclusive relationship (i.e., with possible further variablesinvolved), the term a/an should be intended as one or more items (unlessexpressly indicated otherwise), and the term means for (or anymeans-plus-function formulation) should be intended as any structureadapted or configured for carrying out the relevant function.

For example, an embodiment provides a method for testing performance ofa software product. However, the method may test any type of performance(for example, load, stress, endurance) for any software product (forexample, application, middleware, operating system).

In an embodiment, the method comprises performing one or more completeexecutions of a performance test. However, the complete executions ofthe performance test may be in any number (down to only one, performedonly at the beginning or in any other situations, such as for every newrelease, service pack).

In an embodiment, the complete executions of the performance test are ofcorresponding versions of the software product. However, the versionsmay be of any type (for example, service levels, releases) and in anynumber (with one or more iterations of the complete execution of theperformance test for each version).

In an embodiment, the complete executions of the performance testprovide corresponding results each comprising a value of at least oneperformance parameter after recognizing a steady state thereof. However,the performance parameters may be in any number and of any type (forexample, measuring consumption of any hardware and/or software resource,any responsiveness, different, additional or alternative characteristicsrelating to operation of the software product, such as warming of thecomputer, or any combination thereof). Moreover, the steady state of theperformance trends may be recognized in any way (for example, whenvariation or the first derivative of the performance trend remainswithin a threshold) in any stability time-window.

In an embodiment, the method comprises determining a performance profileof the software product according to the complete executions of theperformance test (which performance profile models a complete trend ofsaid at least one performance parameter during the complete executionsof the performance test). However, the performance profile may bedetermined in any way (for example, by weighting the performance trendsdecreasingly with age).

In an embodiment, the method comprises performing one or more partialexecutions of the performance test of corresponding further versions ofthe software product (with each of the partial executions of theperformance test that has a partial duration preceding said recognizingthe steady state). However, each partial duration may have any length(for example, at least equal to the transient phase or even shorter thanthe transient phase) and may be determined in any way (for example,statically or dynamically, directly or incrementally).

In an embodiment, the method comprises estimating corresponding resultsof the partial executions of the performance test according to theperformance profile from corresponding partial trends of said at leastone performance parameter during the partial executions of theperformance test. However, each result may be estimated in any way (forexample, by any extrapolation techniques, such as of linear, polynomialtype).

In an embodiment, the method comprises updating the performance profileaccording to each of the partial trends. However, the performanceprofile may be updated according to every partial trend, only until aminimum partial duration is determined or never (with the performanceprofile that is determined only according to the complete trends in thelatter case).

In an embodiment, the method comprises determining a change indictor ofa change of each of the versions of the software product with respect toa corresponding previous version of the versions of the softwareproduct. However, the change indicator may be determined in any way (forexample, manually, automatically, semi-automatically).

In an embodiment, the method comprises estimating the result of each ofthe partial executions of the performance test further according to thechange indicator of the corresponding version of the software product.However, the change indicator may be taken into account in any way (forexample, by updating the performance profile accordingly) or thisfeature may be omitted at all.

In an embodiment, said step of determining a change indictor comprisesmeasuring one or more software metrics of each of the versions of thesoftware product indicative of a complexity and/or a size thereof.However, the software metrics may be in any number and may indicate anycomplexity (for example, the number of functions), any size (forexample, the number of modules), different, additional or alternativecharacteristics of the software product (for example, a number ofconnections thereto), or any combination thereof (for example, byaveraging them according to different weights).

In an embodiment, said step of determining a change indictor comprisescalculating the change indicator of each of the versions of the softwareproduct according to a comparison of the software metrics being measuredfor the version of the software product and for the correspondingprevious version of the software product. However, the change indicatormay be calculated according to any comparison (for example, by comparingeach software metric individually and then combining the obtainedresults, by combining the software metrics of each version and thencomparing combinations of the software metrics, according to theaverage, median, peak of the software metrics) or more generally in anyother way (for example, by taking into account any number of precedingversions, possibly weighted decreasingly with age).

In an embodiment, said step of performing one or more partial executionsof the performance test comprises determining the partial duration ofeach of the partial executions of the performance test according to anumber of previous partial executions of said partial executions of theperformance test. However, the partial duration may vary in any wayaccording to the number of previous (total and/or partial) executions ofthe performance test, or even independent of previous executions of theperformance test.

In an embodiment, the method comprises determining the partial durationof each of the partial executions of the performance test according tothe change indicator of the corresponding version of the softwareproduct. However, the partial duration may vary in any way according tothe change indicator (for example, based on the trend of the performancetest, alone or in combination with the number of previous executions),or even independently of it.

In an embodiment, the method comprises (for each of the partialexecutions of the performance test) estimating a performance target forsaid at least one performance parameter according to the performanceprofile in correspondence to the partial duration of the partialexecution of the performance test. However, the performance target mayby estimated in any way (either the same or different with respect tothe estimation of the results).

In an embodiment, the method comprises reducing the partial duration fora next partial execution of said partial executions of the performancetest in response to a match of the partial trend in correspondence tothe partial duration with the performance target. However, the match maybe determined in any way (for example, tolerating any difference, downto none) and the partial duration may be reduced in any way accordingly(for example, with a decreasing pitch, in response to every match oronly after two or more consecutive matches).

In an embodiment, said step of reducing the partial duration comprisesreducing the partial duration at most to a minimum value. However, theminimum value may be of any type (for example, determined statically byany fixed or percentage value, dynamically by any converge condition,alone or in combination). In any case, the possibility is not excludedof reducing the partial duration as far as possible (without any minimumvalue thereof).

In an embodiment, the method comprises estimating the performance targetfor each of the partial executions of the performance test according tothe change indicator of the corresponding version of the softwareproduct. However, the change indicator may be taken into account in anyway (either the same or different with respect to the estimation of theresults) or this feature may be omitted at all.

In an embodiment, the method comprises (for each of the partialexecutions of the performance test) incrementing the partial durationfor the partial execution of the performance test in response to amismatch of the partial trend in correspondence to the partial durationwith the performance target. However, the partial duration may beincremented in any way (either the same or different with respect to itsdecrement), or different, additional or alternative operations may beperformed in response to the mismatch (for example, stopping theperformance test and setting the performance test as failed,re-determining the performance profile from the complete executions ofthe performance test when the failure happens after the minimum value ofthe partial duration has been determined).

In an embodiment, the method comprises (for each of the partialexecutions of the performance test) determining a failure of the partialexecution of the performance test in response to the partial trendreaching the performance target before the partial duration. However,this failure may be determined in any way (for example, tolerating anydifference, down to none, up to any time before the end of the partialduration, down to none). In any case, the departure from the performanceprofile may be monitored in any way, even without any performance target(for example, according to a trend of the difference over time betweenthe partial trend and the performance target) or this feature may beomitted at all.

In an embodiment, the method comprises determining each of the completetrends and each of the partial trends by high-pass filtering values ofthe corresponding performance parameter being measured during thecorresponding complete execution of the performance test or partialexecution of the performance test, respectively. However, the completeand partial trends may be determined in any way, either the same ordifferent (for example, with or without the smoothing of the performancesignal, by curve fitting techniques).

Generally, similar considerations apply if the same solution isimplemented with an equivalent method (by using similar steps with thesame functions of more steps or portions thereof, removing some stepsbeing non-essential, or adding further optional steps). Moreover, thesteps may be performed in a different order, concurrently or in aninterleaved way (at least in part).

An embodiment provides a computer program configured for causing acomputing system to perform the above-mentioned method when the computerprogram is executed on the computing system. An embodiment provides acomputer program product, the computer program product comprising acomputer readable storage medium having program instructions embodiedtherewith, the program instructions being executable by a computingsystem to cause the computing system to perform the same method.However, the computer program may be implemented as a stand-alonemodule, as a plug-in for a pre-existing computer program (for example,the test tool), or even directly in it; moreover, the computer programmay run on any computing system (see below). In any case, the solutionaccording to an embodiment of the present disclosure lends itself to beimplemented even with a hardware structure (for example, by electroniccircuits integrated in one or more chips of semiconductor material), orwith a combination of software and hardware suitably programmed orotherwise configured.

An embodiment provides a system comprising means configured forperforming each of the steps of the above-mentioned method. Anembodiment provides a system comprising a circuitry (i.e., any hardwaresuitably configured, for example, by software) configured for performingeach of the steps of the same method. However, the system may be of anytype (for example, any physical and/or virtual computing machine, eitherstand-alone or communicating with other computing machines via anylocal, wide area, global, cellular or satellite network and exploitingany type of wired and/or wireless connections).

Generally, similar considerations apply if the system has a differentstructure or comprises equivalent components or it has other operativecharacteristics. In any case, every component thereof may be separatedinto more elements, or two or more components may be combined togetherinto a single element, moreover, each component may be replicated tosupport the execution of the corresponding operations in parallel.Moreover, unless specified otherwise, any interactivity betweendifferent components generally does not need to be continuous, and maybe either direct or indirect through one or more intermediaries.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention. The computer readable storage medium can be atangible device that can retain and store instructions for use by aninstruction execution device. The computer readable storage medium maybe, for example, but is not limited to, an electronic storage device, amagnetic storage device, an optical storage device, an electromagneticstorage device, a semiconductor storage device, or any suitablecombination of the foregoing. A non-exhaustive list of more specificexamples of the computer readable storage medium includes the following:a portable computer diskette, a hard disk, a random access memory (RAM),a read-only memory (ROM), an erasable programmable read-only memory(EPROM or Flash memory), a static random access memory (SRAM), aportable compact disc read-only memory (CD-ROM), a digital versatiledisk (DVD), a memory stick, a floppy disk, a mechanically encoded devicesuch as punch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire. Computer readable program instructions described hereincan be downloaded to respective computing/processing devices from acomputer readable storage medium or to an external computer or externalstorage device via a network, for example, the Internet, a local areanetwork, a wide area network and/or a wireless network. The network maycomprise copper transmission cables, optical transmission fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device. Computer readable programinstructions for carrying out operations of the present invention may beassembler instructions, instruction-set-architecture (ISA) instructions,machine instructions, machine dependent instructions, microcode,firmware instructions, state-setting data, or either source code orobject code written in any combination of one or more programminglanguages, including an object oriented programming language such asSmalltalk, C++ or the like, and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention. Aspectsof the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions. These computer readable programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer readable program instructions may also be storedin a computer readable storage medium that can direct a computer, aprogrammable data processing apparatus, and/or other devices to functionin a particular manner, such that the computer readable storage mediumhaving instructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks. Thecomputer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks. The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

A computer program product of the present invention comprises one ormore computer readable hardware storage devices having computer readableprogram code stored therein, said program code executable by one or moreprocessors of a computer system to implement the methods of the presentinvention.

A computer system of the present invention comprises one or moreprocessors, one or more memories, and one or more computer readablehardware storage devices, said one or more hardware storage devicecontaining program code executable by the one or more processors via theone or more memories to implement the methods of the present invention.

In one embodiment, the computer or computer system may be or include aspecial-purpose computer or machine that comprises specialized,non-generic hardware and circuitry (i.e., specialized discretenon-generic analog, digital, and logic based circuitry) for(independently or in combination) particularized for executing onlymethods of the present invention. The specialized discrete non-genericanalog, digital, and logic based circuitry may include proprietaryspecially designed components (e.g., a specialized integrated circuit,such as for example an Application Specific Integrated Circuit (ASIC),designed for only implementing methods of the present invention).

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers or ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method for testing performance of a softwareproduct, said method comprising: performing, by one or more processorsof a computer system, one or more complete executions of a performancetest of corresponding versions of the software product, said completeexecutions of the performance test identifying a steady state value ofeach performance parameter of at least one performance parametermonitored during the performance test after each performance parameterhas transitioned to a steady state during the performance test,determining, by the one or more processors, a performance profile of thesoftware product according to the complete executions of the performancetest, said performance profile modeling a complete trend of the at leastone performance parameter during the complete executions of theperformance test, performing, by the one or more processors, one or morepartial executions of the performance test of corresponding furtherversions of the software product, each partial execution of theperformance test having a partial duration preceding the steady state ofeach performance parameter, and estimating, by the one or moreprocessors, corresponding results of the partial executions of theperformance test according to the performance profile from correspondingpartial trends of the at least one performance parameter during thepartial executions of the performance test.
 2. The method of claim 1,wherein the method comprises: updating, by the one or more processors,the performance profile according to each partial trend of the partialtrends.
 3. The method of claim 1, wherein the method comprises:determining, by the one or more processors, a change indictor of achange of each version of the versions of the software product withrespect to a corresponding previous version of the versions of thesoftware product, and further estimating, by the one or more processors,the result of each partial execution of the partial executions of theperformance test according to the change indicator of the change of thecorresponding version of the software product.
 4. The method of claim 3,wherein said determining the change indictor of the change of eachversion comprises: measuring one or more software metrics of eachversion of the software product indicative of a complexity and/or a sizeof each version, and calculating the change indicator of the change ofeach version of the software product according to a comparison of thesoftware metrics being measured for each version of the software productand for the corresponding previous version of the software product. 5.The method of claim 1, wherein said performing one or more partialexecutions of the performance test comprises: determining the partialduration of each partial execution of the partial executions of theperformance test according to a number of previous partial executions ofthe partial executions of the performance test.
 6. The method of claim1, wherein the method comprises: determining, by the one or moreprocessors, a change indictor of a change of each version of theversions of the software product with respect to a correspondingprevious version of the versions of the software product, anddetermining, by the one or more processors, the partial duration of eachpartial execution of the partial executions of the performance testaccording to the change indicator of the change of the correspondingversion of the software product.
 7. The method of claim 1, wherein themethod comprises, for each partial execution of the performance test:estimating, by the one or more processors, a performance target for theat least one performance parameter according to the performance profilein correspondence to the partial duration of the partial execution ofthe performance test, and reducing, by the one or more processors, thepartial duration for a next partial execution of the partial executionsof the performance test in response to a match of the partial trend, incorrespondence to the partial duration, with the performance target. 8.The method of claim 7, wherein said reducing the partial durationcomprises: reducing the partial duration to no less than a minimum valueof the partial duration.
 9. The method of claim 7, wherein the methodcomprises: determining, by the one or more processors, a change indictorof a change of each version of the versions of the software product withrespect to a corresponding previous version of the versions of thesoftware product, and estimating, by the one or more processors, theperformance target for each partial execution of the partial executionsof the performance test according to the change indicator of the changeof the corresponding version of the software product.
 10. The method ofclaim 7, wherein the method comprises, for each partial execution of thepartial executions of the performance test: incrementing, by the one ormore processors, the partial duration for the partial execution of theperformance test in response to a mismatch of the partial trend, incorrespondence to the partial duration, with the performance target. 11.The method of claim 7, wherein the method comprises, for each of thepartial executions of the performance test: determining, by the one ormore processors, a failure of the partial execution of the performancetest in response to the partial trend reaching the performance targetbefore the partial duration has elapsed.
 12. The method of claim 1,wherein the method comprises: determining, by the one or moreprocessors, each complete trend of the complete trends or each partialtrend of the partial trends by high-pass filtering values of thecorresponding performance parameter being measured during thecorresponding complete execution of the performance test or thecorresponding partial execution of the performance test, respectively.13. A computer program product, comprising one or more computer readablehardware storage devices having computer readable program code storedtherein, said program code containing instructions executable by one ormore processors of a computer system to implement a method for testingperformance of a software product, said method comprising: performing,by the one or more processors, one or more complete executions of aperformance test of corresponding versions of the software product, saidcomplete executions of the performance test identifying a steady statevalue of each performance parameter of at least one performanceparameter monitored during the performance test after each performanceparameter has transitioned to a steady state during the performancetest, determining, by the one or more processors, a performance profileof the software product according to the complete executions of theperformance test, said performance profile modeling a complete trend ofthe at least one performance parameter during the complete executions ofthe performance test, performing, by the one or more processors, one ormore partial executions of the performance test of corresponding furtherversions of the software product, each partial execution of theperformance test having a partial duration preceding the steady state ofeach performance parameter, and estimating, by the one or moreprocessors, corresponding results of the partial executions of theperformance test according to the performance profile from correspondingpartial trends of the at least one performance parameter during thepartial executions of the performance test.
 14. The computer programproduct of claim 13, wherein the method comprises: updating, by the oneor more processors, the performance profile according to each partialtrend of the partial trends.
 15. The computer program product of claim13, wherein the method comprises: determining, by the one or oreprocessors, a change indictor of a change of each version of theversions of the software product with respect to a correspondingprevious version of the versions of the software product, and furtherestimating, by the one or more processors, the result of each partialexecution of the partial executions of the performance test according tothe change indicator of the change of the corresponding version of thesoftware product.
 16. The computer program product of claim 15, whereinsaid determining the change indictor of the change of each versioncomprises: measuring one or more software metrics of each version of thesoftware product indicative of a complexity and/or a size of eachversion, and calculating the change indicator of the change of eachversion of the software product according to a comparison of thesoftware metrics being measured for each version of the software productand for the corresponding previous version of the software product. 17.A computer system, comprising one or more processors, one or morememories, and one or more computer readable hardware storage devices,said one or more hardware storage device containing program codeexecutable by the one or more processors via the one or more memories toimplement a method for testing performance of a software product, saidmethod comprising: performing, by the one or more processors, one ormore complete executions of a performance test of corresponding versionsof the software product, said complete executions of the performancetest identifying a steady state value of each performance parameter ofat least one performance parameter monitored during the performance testafter each performance parameter has transitioned to a steady stateduring the performance test, determining, by the one or more processors,a performance profile of the software product according to the completeexecutions of the performance test, said performance profile modeling acomplete trend of the at least one performance parameter during thecomplete executions of the performance test, performing, by the one ormore processors, one or more partial executions of the performance testof corresponding further versions of the software product, each partialexecution of the performance test having a partial duration precedingthe steady state of each performance parameter, and estimating, by theone or more processors, corresponding results of the partial executionsof the performance test according to the performance profile fromcorresponding partial trends of the at least one performance parameterduring the partial executions of the performance test.
 18. The computersystem of claim 17, wherein the method comprises: updating, by the oneor more processors, the performance profile according to each partialtrend of the partial trends.
 19. The computer system of claim 17,wherein the method comprises: determining, by the one or moreprocessors, a change indictor of a change of each version of theversions of the software product with respect to a correspondingprevious version of the versions of the software product, and furtherestimating, by the one or more processors, the result of each partialexecution of the partial executions of the performance test according tothe change indicator of the change of the corresponding version of thesoftware product.
 20. The computer system of claim 19, wherein saiddetermining the change indictor of the change of each version comprises:measuring one or more software metrics of each version of the softwareproduct indicative of a complexity and/or a size of each version, andcalculating the change indicator of the change of each version of thesoftware product according to a comparison of the software metrics beingmeasured for each version of the software product and for thecorresponding previous version of the software product.